{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f1f51cd-6d93-464e-8f4a-41666c202d51",
   "metadata": {},
   "source": [
    "## 使用pdfplumber解析PDF文件\n",
    "### 项目简介\n",
    "[pdfplumber](https://github.com/jsvine/pdfplumber)项目（基于pdfminer.six开发）支持解析PDF文件，获取每个文本字符、矩形、和线条的详细信息，还支持表格提取和可视化调试。\n",
    "\n",
    "对于机器生成的pdf效果更佳，不适用于扫描文件PDF\n",
    "\n",
    "支持：Python 3.8-3.11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f5f34b66-2c12-422b-9f3c-6d0b9325928c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# !pip install -r ../requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "333fdf7d-aa30-40be-af6e-cb8022cf8f30",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pdfplumber\n",
    "import pandas as pd\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "300c9576-0512-42bd-86ce-1f7c871fcf57",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'CreationDate': \"D:20060717205532+08'00'\",\n",
       " 'Subject': 'For Personal Learning!',\n",
       " 'Author': 'Asiaing.com',\n",
       " 'Creator': 'PScript5.dll Version 5.2',\n",
       " 'Producer': 'Acrobat Distiller 7.0.5 (Windows)',\n",
       " 'ModDate': \"D:20060717210222+08'00'\",\n",
       " 'Title': 'Hemingway, Ernest - The Old Man and the Sea'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# 加载文件，可以是PDF文件路径、作为字节加载的文件对象、作为字节加载的类似文件的对象\n",
    "pdf = pdfplumber.open(\"../data/The_Old_Man_of_the_Sea.pdf\")\n",
    "# 文件的元数据信息\n",
    "pdf.metadata\n",
    "# 文件页数\n",
    "pdf.pages\n",
    "\n",
    "type(pdf.pages[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5bf5739f-e939-49e6-918e-ae127c9aa437",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Page:1>, <Page:2>]\n"
     ]
    }
   ],
   "source": [
    "pdf = pdfplumber.open(\"../data/test.pdf\")\n",
    "pages = pdf.pages\n",
    "print(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c459fd48-de3b-48ed-bd13-f3141765cd09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "595 842\n"
     ]
    }
   ],
   "source": [
    "# 输出pdf的页码，从1开始\n",
    "print(pages[0].page_number)\n",
    "# 输出pdf的宽高\n",
    "print(pages[0].width, pages[0].height)\n",
    "# .objects/chars/lines/rects/curves/images：返回页面嵌入的每个类型对象的一个字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24e19faf-04fc-4f73-a469-5134b7de0b90",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09013c27-a90c-4883-abbe-c8a3f5cf0c30",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28efec18-b7a5-4fc0-9e5f-e52419882c3f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5fbcc8bc-e8d7-4542-81ea-65c20accba03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Test Data  \\nThis dataset contains two test samples provided by ChatGPT, an AI language model by OpenAI. \\nThese samples include a markdown table and an English text passage, which can be used to test an \\nEnglish-to-Chinese translation software supporting both text and table formats.\\nText testing  \\nThe quick brown fox jumps over the lazy dog. This pangram contains every letter of the English \\nalphabet at least once. Pangrams are often used to test fonts, keyboards, and other text-related \\ntools. In addition to English, there are pangrams in many other languages. Some pangrams are more \\ndiﬃcult to construct due to the unique characteristics of the language.\\nTable Testing  \\nFruit Color Price (USD)\\nApple Red 1.20\\nBanana Yellow 0.50\\nOrange Orange 0.80\\nStrawberry Red 2.50\\nBlueberry Blue 3.00\\nKiwi Green 1.00\\nMango Orange 1.50\\nGrape Purple 2.00'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 获取页面所有的字符对象，汇集成单一字符串\n",
    "# layout=False, x_tolerance=3, y_tolerance=3：layout=False时，在一个字符的x1和下一个字符的x0之间的差异大于x_tolerance时添加空格，两个字符的doctop之间差异大于y_tolerance时添加换行符；\n",
    "# x_density=7.25, y_density=13, **kwargs：layout=True时，尝试模仿页面上文本的结构布局，使用x_density和y_density来确定每个点的最小字符、换行符数量，所有剩余kwargs都会传递给extract_word(...)，是计算的第一步\n",
    "pages[0].extract_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6302fd2a-3a2a-45e2-86b1-2ea904e76220",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "        Test    Data                                                              \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "        This dataset contains two test samples provided by ChatGPT, an AI language model by OpenAI.\n",
      "        These samples include a markdown table and an English text passage, which can be used to test an\n",
      "        English-to-Chinese translation software supporting both text and table formats.\n",
      "                                                                                  \n",
      "                                                                                  \n",
      "        Text  testing                                                             \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "        The quick brown fox jumps over the lazy dog. This pangram contains every letter of the English\n",
      "        alphabet at least once. Pangrams are often used to test fonts, keyboards, and other text-related\n",
      "        tools. In addition to English, there are pangrams in many other languages. Some pangrams are more\n",
      "                                                                                  \n",
      "        difficult to construct due to the unique characteristics of the language. \n",
      "                                                                                  \n",
      "        Table  Testing                                                            \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "          Fruit                  Color            Price (USD)                     \n",
      "                                                                                  \n",
      "          Apple                  Red              1.20                            \n",
      "                                                                                  \n",
      "          Banana                 Yellow           0.50                            \n",
      "                                                                                  \n",
      "          Orange                 Orange           0.80                            \n",
      "                                                                                  \n",
      "          Strawberry             Red              2.50                            \n",
      "                                                                                  \n",
      "          Blueberry              Blue             3.00                            \n",
      "                                                                                  \n",
      "          Kiwi                   Green            1.00                            \n",
      "                                                                                  \n",
      "          Mango                  Orange           1.50                            \n",
      "                                                                                  \n",
      "          Grape                  Purple           2.00                            \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n",
      "                                                                                  \n"
     ]
    }
   ],
   "source": [
    "# 保留文本布局，能够识别表格\n",
    "print(pages[0].extract_text(layout=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "d4d65e80-8df1-438f-93e2-43bc238b3fe4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[['Fruit', 'Color', 'Price (USD)'],\n",
       "  ['Apple', 'Red', '1.20'],\n",
       "  ['Banana', 'Yellow', '0.50'],\n",
       "  ['Orange', 'Orange', '0.80'],\n",
       "  ['Strawberry', 'Red', '2.50'],\n",
       "  ['Blueberry', 'Blue', '3.00'],\n",
       "  ['Kiwi', 'Green', '1.00'],\n",
       "  ['Mango', 'Orange', '1.50'],\n",
       "  ['Grape', 'Purple', '2.00']]]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 获取单页表格\n",
    "p1_table = pages[0].extract_tables()\n",
    "print(p1_table)\n",
    "\n",
    "# 获取但也所有表格\n",
    "# tables = pages[0].extract_tables()\n",
    "# len(tables)\n",
    "# tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "c5723176-ae0f-460e-b095-27a3285ec4c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(p1_table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "165d1a0d-2f70-4457-9a75-298956778e9a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pdfplumber.table.TableFinder'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<pdfplumber.table.Table at 0x1e0dbcdb310>]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 获取\n",
    "# debug_table = pages[0].debug_tablefinder()\n",
    "# print(type(debug_table))\n",
    "# debug_table.tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "51062457-b344-4316-a4f3-06c1b7addcb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "df = pd.DataFrame(p1_table)\n",
    "df\n",
    "print(len(df))\n",
    "print(len(df.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e31a71c0-d1a3-4334-9f27-6a7d4e2e0735",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Fruit</th>\n",
       "      <th>Color</th>\n",
       "      <th>Price (USD)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Apple</th>\n",
       "      <th>Red</th>\n",
       "      <th>1.20</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Banana</th>\n",
       "      <th>Yellow</th>\n",
       "      <th>0.50</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Orange</th>\n",
       "      <th>Orange</th>\n",
       "      <th>0.80</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Strawberry</th>\n",
       "      <th>Red</th>\n",
       "      <th>2.50</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Blueberry</th>\n",
       "      <th>Blue</th>\n",
       "      <th>3.00</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Kiwi</th>\n",
       "      <th>Green</th>\n",
       "      <th>1.00</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Mango</th>\n",
       "      <th>Orange</th>\n",
       "      <th>1.50</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Grape</th>\n",
       "      <th>Purple</th>\n",
       "      <th>2.00</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [(Fruit, Apple, Banana, Orange, Strawberry, Blueberry, Kiwi, Mango, Grape), (Color, Red, Yellow, Orange, Red, Blue, Green, Orange, Purple), (Price (USD), 1.20, 0.50, 0.80, 2.50, 3.00, 1.00, 1.50, 2.00)]\n",
       "Index: []"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用pandas来存储表格数据，内容为1~末行，首行为pandas的DF标题\n",
    "df = pd.DataFrame(p1_table[1:], columns=p1_table[0])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50866d62-f739-45e8-b102-c99562c39ce1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6888707-5ed4-410a-bdc8-06a40e650504",
   "metadata": {},
   "outputs": [],
   "source": [
    "# PDF转换为图片\n",
    "pages[0].to_image()\n",
    "pages[1].to_image()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "78013d9b-d685-4963-b87f-90d9a3dbf1ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从页面获取各个图像分辨率参数\n",
    "pages[1].images\n",
    "# 获取第一个图像的的参数，并打包成元组\n",
    "img = pages[1].images[0]\n",
    "bbox = (img[\"x0\"], img[\"top\"], img[\"x1\"], img[\"bottom\"])\n",
    "\n",
    "# 可视化剪裁后的第二页\n",
    "cropped_page = pages[1].crop(bbox)\n",
    "im = cropped_page.to_image()\n",
    "# 抗锯齿\n",
    "im = cropped_page.to_image(antialias=True)\n",
    "# 1080分辨率\n",
    "im = cropped_page.to_image(antialias=True, resolution=1080)\n",
    "im.save(\"./images/test_p2_image_1080.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59b7ee5a-de61-4485-9ef0-51814899436a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56449e4f-dc48-454b-a671-5e96e28bfb52",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbac60e5-0876-4695-90a4-d209c340d3ac",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "c145ee21-7428-4b6d-8c8d-42d8bcf68a42",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     水果    颜色  价格 (美元)\n",
      "苹果      红色         1.2\n",
      "香蕉      黄色         0.5\n",
      "橙子      橙色         0.8\n",
      "草莓      红色         2.5\n",
      "蓝莓      蓝色         3.0\n",
      " 猕猴桃    绿色         1.0\n",
      "芒果      橙色         1.5\n",
      "葡萄      紫色         2.0\n"
     ]
    }
   ],
   "source": [
    "from io import StringIO\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "table = '''| 水果     | 颜色   | 价格 (美元) |\n",
    "|----------|--------|--------------|\n",
    "| 苹果     | 红色   | 1.20         |\n",
    "| 香蕉     | 黄色   | 0.50         |\n",
    "| 橙子     | 橙色   | 0.80         |\n",
    "| 草莓     | 红色   | 2.50         |\n",
    "| 蓝莓     | 蓝色   | 3.00         |\n",
    "| 猕猴桃   | 绿色   | 1.00         |\n",
    "| 芒果     | 橙色   | 1.50         |\n",
    "| 葡萄     | 紫色   | 2.00         |'''\n",
    "\n",
    "clean_data = \"\\n\".join(line for line in table.split(\"\\n\") if not set(line) <= {\"|\", \"-\", \" \"})\n",
    "\n",
    "# 2. 解析成 DataFrame\n",
    "df = pd.read_csv(StringIO(clean_data), sep=\"|\", skipinitialspace=True)\n",
    "\n",
    "# 3. 清理列名和数据\n",
    "df.columns = [col.strip() for col in df.columns]  # 去除列名空格\n",
    "# df = df.drop(columns=[\"\"])  # 删除可能的空列\n",
    "# df[\"价格（美元）\"] = df[\"价格（美元）\"].astype(float)  # 转换价格列为 float\n",
    "df = df.dropna(axis=1, how=\"all\")  # 删除所有值均为空的列\n",
    "df = df.reset_index(drop=True)\n",
    "\n",
    "print(df.to_string(index=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "f455843f-f0d9-4525-a80f-fc21c585e0c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Unnamed: 0', '----------', '--------', '--------------', 'Unnamed: 4'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a50cc54-b13d-42d4-9056-941538bd8daf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langChain",
   "language": "python",
   "name": "langchain"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
